Parameters reference

Complete reference for ik_llama.cpp command-line parameters

Complete reference for llama-server, llama-cli, llama-sweep-bench, llama-bench, llama-imatrix, and llama-quantize parameters. All parameters supported by llama-server can also be used with the other tools where applicable.

LLM jargon

Common terms used throughout this documentation and in model descriptions.

Term	Meaning
LLM / model	Large Language Model trained on vast amounts of text using machine learning.
Tensors	The foundational building block of a model — a multi-dimensional array of numbers (scalar, vector, matrix, or higher-dimensional).
Layers	Modular units stacked to form the network, each transforming the input tensors in some way.
Weights	Numerical values associated with connections between tensors in each layer.
Activations	Output of a layer after it has performed its computations.
FA	Flash Attention — an efficient transformer attention algorithm.
VRAM	Dedicated memory on the GPU.
Inference	Running a model to generate responses.
GGUF	The file format used by ik_llama.cpp and llama.cpp.
Quants	Compressed model formats that reduce precision to save space and improve speed.
BPW	Bits per weight — measures the compression ratio of a quant.
imatrix	Importance matrix generated from calibration text; improves quantization quality.
Model splits	A GGUF file split into multiple parts for easier upload/download. Specify only the first part when loading.
PP	Prompt processing — encoding the input tokens.
TG	Token generation — producing the output tokens one by one.
t/s	Tokens per second — measures PP and TG speed.
Full GPU	All tensors and computation offloaded to the GPU.
Hybrid CPU/GPU	Partial offload — some tensors in VRAM, others in RAM.

General parameters

Core parameters for loading and running any model.

Parameter	Description	Default	Notes
-h, --help, --usage	Print usage and exit	—	—
--fit	Automatically fit to available VRAM	off	Loads as many tensors to GPU as VRAM permits. Cannot be used with --cpu-moe, --n-cpu-moe, or tensor overrides.
--fit-margin N	Safety VRAM margin in MiB when using --fit	1024	Increase if you get CUDA OOM during model load.
-t, --threads N	Threads for token generation	4	Match the number of physical CPU cores. Avoid odd numbers.
-tb, --threads-batch N	Threads for batch/prompt processing	Same as --threads	For full GPU offload, use a lower number (e.g. 2).
-c, --ctx-size N	Context size (prompt + generation)	0 (from model)	Determines KV cache size. With parallel slots, this is split across all slots.
-n, --predict N	Max tokens to generate	-1 (infinity)	-2 = fill context. Safe to leave at default.
-b, --batch-size N	Logical maximum batch size	2048	Higher values may improve t/s on GPU at the cost of memory.
-ub, --ubatch-size N	Physical maximum batch size	512	Similar effect to --batch-size.
--keep N	Tokens to keep from initial prompt	0	-1 = keep all.
--chunks N	Max chunks to process	-1 (all)	—
-fa, --flash-attn	Enable Flash Attention	on	Improves t/s and reduces memory usage. Use auto/on/off.
--no-fa, --no-flash-attn	Disable Flash Attention	—	Alternative to -fa off.
-mla, --mla-use	Enable MLA	3	0/1/2/3. For DeepSeek and other MLA models.
-amb, --attention-max-batch	Max batch size for attention	0	Specifies maximum K*Q size in MB to tolerate.
-fmoe, --fused-moe	Fuse ffn_up and ffn_gate in MoE	—	Speedup for MoE models.
--no-fmoe, --no-fused-moe	Disable fused MoE	Enabled	See --fused-moe.
-ger, --grouped-expert-routing	Enable grouped expert routing	Disabled	For BailingMoeV2 architecture (Ling/Ring models).
--no-fug, --no-fused-up-gate	Disable fused up-gate	Enabled	Turns off the up-gate speedup for dense models.
--no-mmad, --no-fused-mul-multiadd	Disable fused mul-multi_add	Enabled	—
-gr, --graph-reuse	Enable graph reuse	Enabled	For models with fast TG (100+ t/s).
--no-gr, --no-graph-reuse	Disable graph reuse	Disabled	—
-ser, --smart-expert-reduction	Expert reduction Kmin,t	-1, 0	Use fewer active experts. -ser 1,6 uses exactly 6 experts.
-mqkv, --merge-qkv	Merge Q, K, V projections	0	Downside: mmap cannot be used.
-muge, --merge-up-gate-experts	Merge ffn_up/gate_exps	0	Speedup on some models.
-khad, --k-cache-hadamard	Hadamard transform for K-cache	0	May improve quality at low KV quantization levels.
-sas, --scheduler_async	Async evaluation of compute graphs	0	—
-vq, --validate-quants	Validate quantized data on load	0	Reports NaN tensors in the loaded model.
-sp, --special	Enable special token output	false	—
--no-warmup	Skip empty warmup run	—	—
--mlock	Keep model in RAM (no swap)	—	—
--no-mmap	Disable memory-mapped model loading	—	Slower load but may reduce pageouts.
-rtr, --run-time-repack	Repack tensors to interleaved format	—	ik_llama.cpp exclusive. May improve performance.
--ctx-checkpoints N	Checkpoints per slot	—	For recurrent models (Qwen3-Next, Qwen3.5-MoE).
--ctx-checkpoints-interval N	Min tokens between checkpoints	—	Smaller values = more frequent checkpoints during PP.

Speculative decoding

Speculative decoding accelerates generation by using a fast draft model to predict multiple tokens ahead, which the main model then verifies in a single forward pass.

Parameter	Description	Default	Notes
-td, --threads-draft N	Threads for draft model generation	Same as --threads	—
-tbd, --threads-batch-draft N	Threads for draft model batch processing	Same as --threads-draft	—
-ps, --p-split N	Speculative decoding split probability	0.1	—
-cd, --ctx-size-draft N	Context size for draft model	0 (from model)	Similar to --ctx-size but for the draft model.
-ctkd, --cache-type-k-draft TYPE	KV cache K type for draft model	—	See -ctk.
-ctvd, --cache-type-v-draft TYPE	KV cache V type for draft model	—	See -ctv.
-draft, --draft-params	Comma-separated draft model parameters	—	—
--spec-ngram-size-n N	ngram lookup size N	12	For ngram-simple/ngram-map speculative decoding.
--spec-ngram-size-m N	ngram draft size M	48	For ngram-simple/ngram-map speculative decoding.
--spec-ngram-min-hits N	Min hits for ngram-map	1	—
--spec-type Name	Speculative decoding type	—	none, ngram-cache, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod.
-mtp, --multi-token-prediction	Enable MTP decoding	—	For GLM-4.x MoE models.
-no-mtp, --no-multi-token-prediction	Disable MTP decoding	—	—
--draft-max	Max draft tokens	—	For MTP decoding.
--draft-p-min	Min draft probability	—	For MTP decoding.

Cache prompt to host memory

When a conversation ends, its KV cache is saved to RAM and can be restored when the same or similar prompt is seen again. This greatly reduces prompt processing time when switching between conversations.

If available RAM is very limited, disable this with -cram 0 to avoid memory swapping.

Parameter	Description	Default	Notes
-cram, --cache-ram N	Maximum cache size in MiB	8192	-1 = no limit. 0 = disable. Especially useful for coding agents that re-send similar prompts.
-crs, --cache-ram-similarity N	Similarity threshold to trigger cache reuse	0.50	—
-cram-n-min, --cache-ram-n-min N	Min cached tokens to trigger cache reuse	0	—

Sampling

Sampling controls how tokens are selected during generation. The default sampler pipeline provides a good balance for most use cases.

For a detailed overview of sampling techniques, see the llm_samplers_explained guide.

Parameter	Description	Default	Notes
--samplers SAMPLERS	Ordered sampler pipeline (semicolon-separated)	dry;top_k;tfs_z;typical_p;top_p;min_p;xtc;top_n_sigma;temperature;adaptive_p	Example: --samplers min_p;temperature
--sampling-seq SEQUENCE	Shorthand sampler sequence	dkfypmxntw	Same as --samplers in abbreviated form.
--banned-string-file	File containing banned output strings (one per line)	—	—
--banned-n	Number of tokens banned during rewind	-1	-1 = all tokens.

Prompt template

The prompt template controls how chat messages are formatted before being sent to the model. An incorrect template can significantly degrade output quality.

Parameter	Description	Default	Notes
--jinja	Use Jinja template from model metadata	Template from model	Required for function/tool calling.
--chat-template JINJA_TEMPLATE	Override chat template inline	Disabled	Use --chat-template chatml as a fallback when no official tool_use template exists.
--chat-template-file FILE	Load chat template from file	—	Useful when the GGUF metadata contains a buggy template — download only the fixed .jinja file instead of re-downloading the full model.
--reasoning-format FORMAT	Control reasoning/think tag handling	none	none: leave thoughts in message.content. deepseek: move thoughts to message.reasoning_content. deepseek-legacy: keep tags in content AND populate reasoning_content.
--chat-template-kwargs JSON	Additional params for the Jinja template parser	—	Example: --chat-template-kwargs '{"reasoning_effort": "medium"}'
--reasoning-budget N	Max thinking tokens allowed	-1 (unrestricted)	0 = disable thinking.
--reasoning-tokens FORMAT	Exclude reasoning tokens for slot selection	auto	—

Context hacking (KV cache)

The KV cache stores past attention computations to avoid reprocessing tokens. These parameters control where the cache lives and how it is quantized.

The KV cache is stored on the same device as the associated attention tensors. Quantizing the KV cache can significantly reduce VRAM usage.

Parameter	Description	Default	Notes
-dkvc, --dump-kv-cache	Verbose KV cache debug output	—	—
-nkvo, --no-kv-offload	Keep KV cache on CPU	—	Frees VRAM but reduces prompt processing speed.
-ctk, --cache-type-k TYPE	KV cache data type for K	f16	Reduces K size; may slightly affect quality. Requires Flash Attention.
-ctv, --cache-type-v TYPE	KV cache data type for V	f16	See -ctk. K-cache usually needs higher quality than V-cache.
--no-context-shift	Disable context shift	—	—
--context-shift	Configure context shift	on	auto/on/off/0/1. Slides the KV window when context is full.

KV cache types (build with -DGGML_IQK_FA_ALL_QUANTS=ON for the full list):

Type	Notes
f16	Default. Full precision.
q8_0	Half the size, minimal quality loss.
q8_KV	Fast ik_llama.cpp-specific 8-bit KV type.
q6_0	Good quality/size balance.
bf16	Available on CPUs with native BF16 support.

Parallel processing

Serve multiple users or frontends simultaneously. The WebUI uses parallel slots to allow starting a new chat while another is still generating.

Parameter	Description	Default	Notes
-np, --parallel N	Number of parallel decode slots	1	The total --ctx-size is divided across all slots.

GPU offload

ik_llama.cpp provides extensive control over what runs on the GPU. For a full guide, see GPU offloading and Hybrid CPU/GPU inference.

Parameter	Description	Default	Notes
-ngl, --gpu-layers N	Layers to store in VRAM	—	Use 999 to offload everything. For MoE, use more than the model layer count.
-ngld, --gpu-layers-draft N	Layers for draft model in VRAM	—	See -ngl.
--cpu-moe	Keep all MoE expert weights in RAM	—	Simple one-flag hybrid mode for MoE.
--n-cpu-moe N	Keep first N layers’ MoE weights in RAM	—	Useful when some VRAM is available for experts.
-sm, --split-mode MODE	Multi-GPU split strategy	none	none: single GPU. layer: split by layer. graph: split computation graph (best for mixed GPU setups).
-ts, --tensor-split SPLIT	VRAM fraction per GPU (comma-separated)	—	Example: -ts 3,1 gives 75% to GPU 0, 25% to GPU 1.
-dev, --device LIST	Specific GPU devices to use	—	Example: -dev CUDA0,CUDA1.
-devd, --device-draft LIST	GPU devices for draft model	—	—
-mg, --main-gpu i	GPU index for single-GPU mode	—	Used with -sm none.
-ot, --override-tensor REGEX=DEVICE	Place tensors by regex	—	Example: \.ffn_.*_exps\.=CPU. Can be specified multiple times.
-op, --offload-policy a,b	Per-operation offload control	—	a = GGML op enum value, b = 0 (CPU) or 1 (GPU). -op -1,0 disables all GPU offload.
-ooae, --offload-only-active-experts	Offload only activated MoE experts	ON	Reduces RAM→VRAM transfer for sparse models.
-no-ooae	Disable active-expert-only offload	—	May help when large batches activate most experts.
--fit	Auto-fit tensors to available VRAM	off	Cannot be combined with --cpu-moe, --n-cpu-moe, or -ot.
--fit-margin N	VRAM safety margin for --fit (MiB)	1024	Increase if CUDA OOM occurs during load.
-grt, --graph-reduce-type TYPE	Data type for inter-GPU transfers	f32	q8_0/bf16/f16/f32. Lower precision = less bandwidth used.
--max-gpu N	Max GPUs per layer with graph split	—	Useful when using all GPUs hurts performance.
-cuda, --cuda-params LIST	CUDA-specific tuning parameters	—	Controls fusion, offload threshold, MMQ-ID threshold. Example: -cuda graphs=0.
-cuda fa-offset=VALUE	FP16 precision offset for FA	0	Fix FP16 overflow in FA at very long contexts. Value in [0..3].
-smgs, --split-mode-graph-scheduling	Force graph scheduling in split mode	0	—

Model options

Parameters for configuring how the model is loaded and how draft models work.

Parameter	Description	Default	Notes
-m, --model FNAME	Path to model GGUF file	models/$filename	Required. For split models, specify only the first part.
-md, --model-draft FNAME	Draft model for speculative decoding	unused	—
--draft-max, --draft, --draft-n N	Max draft tokens for speculative decoding	16	—
--draft-min, --draft-n-min N	Min draft tokens	—	—
--draft-p-min P	Min speculative decoding probability	0.8	—
--check-tensors	Validate tensor data on load	false	—
--override-kv KEY=TYPE:VALUE	Override model metadata	—	Types: int, float, bool, str. Example: --override-kv tokenizer.ggml.add_bos_token=bool:false.

Server options

Parameters specific to llama-server.

Parameter	Description	Default	Notes
--host HOST	IP address to listen on	127.0.0.1	Use 0.0.0.0 for network access. Never expose to the internet without authentication.
--port PORT	Port to listen on	8080	—
--webui NAME	Which WebUI to serve	auto	none: disabled. auto: default. llamacpp: classic llama.cpp UI.
--api-key KEY	API authentication key	none	Clients must supply this via Authorization: Bearer.
-a, --alias NAME	Model name alias for the API	none	Useful when clients expect a specific model name.

Other tools

sweep-bench

Benchmarks prompt processing and token generation across a sweep of batch sizes. The KV cache is not cleared between runs, so the N_KV column shows how many tokens were in cache.

llama-sweep-bench \
  -m /models/model.gguf \
  -c 12288 -ub 512 \
  -rtr -fa \
  -ctk q8_0 -ctv q8_0

Parameter	Description	Default
-nrep N, --n-repetitions N	Number of repetitions at zero context	—
-n N	Number of TG tokens	ubatch/4

---

llama-bench

Standard benchmark utility.

llama-bench -tgb 4,16 -p 512 -n 128 [other_args]

Parameter	Description	Default
-tgb, --threads-gen-batch	Different thread count for generation vs batch processing	—

---

llama-imatrix

Generate an importance matrix from calibration text. The imatrix improves quantization quality across all quant types.

llama-imatrix \
  -m /models/model-bf16.gguf \
  -f /data/calibration_data_v5_rc.txt \
  -o /models/model.imatrix

Parameter	Description	Default
--layer-similarity, -lsim	Collect activation change statistics using cosine similarity	—
--hide-imatrix	Anonymize the imatrix data file	—

Notes:

• Use convert_imatrix_gguf_to_dat.py to convert GGUF imatrix files to the format used internally.
• imatrix calculation supports models with merged ffn_up/gate_exps tensors.

---

llama-quantize

Quantize a BF16 or F16 model to a compressed format.

llama-quantize \
  --imatrix /models/model.imatrix \
  /models/model-bf16.gguf \
  /models/model-IQ4_NL.gguf \
  IQ4_NL

To split the output for easier distribution:

llama-gguf-split \
  --split --split-max-size 1G \
  --no-tensor-first-split \
  /models/model-IQ4_NL.gguf \
  /models/parts/model-IQ4_NL.gguf

Parameter	Description	Default
--custom-q "regex1=type1,regex2=type2..."	Custom per-tensor quantization rules using regex	—
--dry-run	Print tensor types and sizes without running quantization	—
--partial-requant	Only quantize missing split files in the destination directory	—

Build arguments

CMake build configuration flags.

cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

Argument	Notes
-DGGML_NATIVE=ON	Optimize for the host CPU. Turn off when cross-compiling.
-DGGML_CUDA=ON	Build with CUDA support.
-DCMAKE_CUDA_ARCHITECTURES=86	Target a specific CUDA compute capability (e.g. 86 for RTX 3x00).
-DGGML_ARCH_FLAGS="-march=armv8.2-a+dotprod+fp16"	Pass architecture flags directly.
-DGGML_RPC=ON	Build the RPC backend.
-DGGML_IQK_FA_ALL_QUANTS=ON	Enable all KV cache quantization types.
-DLLAMA_SERVER_SQLITE3=ON	Enable SQLite3 support (for mikupad).
-DCMAKE_TOOLCHAIN_FILE=[...]	Specify a CMake toolchain file (e.g. for Windows + SQLite3).
-DGGML_NCCL=OFF	Disable NCCL.

Environment variables

Environment variables that influence runtime behavior.

CUDA_VISIBLE_DEVICES=0,2 llama-server -m /models/model-bf16.gguf

Variable	Notes
CUDA_VISIBLE_DEVICES	Restrict which GPUs are visible. Example: 0,2 uses the first and third GPU.